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Abstract — We prove that the Bethe expression for the con- 
ditional input-output entropy of cycle LDPC codes on binary 
symmetric channels above the MAP threshold is exact in the large 
block length limit. The analysis relies on methods from statistical 
physics. The finite size corrections to the Bethe expression are 
expressed through a polymer expansion which is controlled 
thanks to expander and counting arguments. 

I. Introduction 

A few years ago Cherktov and Chernyak fH devised a loop 
series which represents the partition function of a general 
vertex model as the product of the Bethe mean field expression 
and a residual partition function over a system of loops. In this 
representation all quantities are entirely expressible in terms 
of Belief Propagation (BP) marginals or messages. However 
it has not been clear so far if this representation leads to 
a controlled series expansions for the log-partition, in other 
words the free energy. If this is the case it should hopefully 
allow to control the difference between the true free energy 
and the Bethe free energy. 

The loop expansion has a potential interest in coding theory 
since Low-Density-Parity-Check (LDPC) and Low-Density- 
Generator-Matrix (LDGM) codes on general binary-input 
memoryless symmetric (BMS) channels fit in the framework of 
(generalized) vertex models. In this context free energy is just 
another name for conditional input-output Shannon entropy. 
For these models it is believed that the Bethe formula for 
the conditional entropy/free energy is exact. However there is 
no general proof, except for the cases of the binary erasure 
channel [T|, LDGM codes for high noise, and in special 
situations for LDPC codes at low noise dSJ. 

We consider cycle LDPC codes for high noise (above the 
MAP threshold) on the binary symmetric channel (BSC). We 
show that, under the assumption that there exists a fixed point 
for the BP equations, the average conditional entropy/free 
energy is given by the Bethe expression. The novelty of the 
approach is to turn the loop expansion into a rigorous tool al- 
lowing to derive provably convergent polymer expansions 101 . 
Controlling the loop expansion is a non-trivial task because in 
most situations of interest the number of loops proliferates. For 
example, this is the case (for the system of fundamental cycles) 
in capacity approaching codes even under MAP decoding jS] . 

II. Loop and polymer representations 

Let r — (y, E) be a graph with vertices a £ V" of regular 
degree d and edges ah G E. The symbol da denotes the set of 
d neighbors of a. In vertex models the degrees of freedom are 



spins (Jab G {^li +1} attached to each edge. At each function 
node a e y we attach a non-negative function fa{<Jda) 
depending only on neighboring variables aQa = {<^ab)b£da- 
We study probability distributions which can be factorized as 

and their associated free energy /„ = ^ InZp. 

For each edge ab SI E we introduce two directed "mes- 
sages" rja-^b and rjb^a- For the moment these variables are 
arbitrary and are collectively denoted by fj. One has the 
identity UJ 



/„ = - In 2'Bctho(?7) + - In Zcorriff). 
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The first term is the Bethe free energy functional, 
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The "partition function" in the second term can be expressed 
as a sum over all subgraphs (not necessarily connected) <? C F 

Zcorr iv) = E ^(9) 

and Kig) = Uaeg ^ith 

ffa b£dar\g 

fa {Soa) Ylbeda e'"" 



Pa (0-9a) 



fa{^da)llb^Oa' 

It is well known that the stationary points of (O satisfy the 
BP fixed point equations. 
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Remarkably, for any solution of Q, K{g) = if g contains a 
degree one node. Thus if rf is a fixed point of the BP equations 
then Zcoir(??) is given by the sum in (|4| over 5 C F with no 
degree one nodes. Such graphs are called loops (see figure [1]). 

One can recognize that Zcorr can be interpreted as the 
partition function of a system of polymers. A loop 5 C F can 
be decomposed into its connected parts in a unique way as 



illustrated on figure [T] Connected loops are called polymers 
and are generically denoted by the letter 7. The important 
point is that by definition the polymers do not intersect. For 
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Fig. 1. Left: an example of a loop graph g with no dangling edge. Right: 
decomposition of g into its connected parts 7^. 

each polymer 7 we define a weight (also called activity), 
^''^(7) = Uae-r "^^^ 9 = ^iLili- Since the 7^ are disjoint, 
riaeg ^a. — 11^=1 ^{li)- Thus equation (|4|i can be cast in the 
form 

1 ^ 

z,o„ {rf)^Y.w\ ^ n ^ (^■') n 11 n 7, = 0) . 

M>0 '7l>-->7Mj=l i<j 

(6) 

In this sum each 7^ runs over all connected subgraphs with no 
dangling edges of the underlying graph F. The sum over the 
number of polymers M has a finite number of terms because 
the polymers cannot intersect. 

In the next paragraphs F is a random d-regular graph. We 
denote by P and E the relevant probability and expectation 
over this ensemble. 

III. Polymer expansion 

We wish to compute the correction to the Bethe free energy 
in (|2]l, namely fcorriv) = -ln^corr(^) when 77 is a BP 
fixed point. Using (|6]l the logarithm can be expanded as a 
power series in K{jiys. This yields the polymer (or Mayer) 
expansion ||4l 

00 ^ M 
M=l 7i,..,7Afi=l 

X E n {-I (7. n 7,^0)}. (7) 
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The third sum is over the set Qm of all connected graphs 
with M vertices labeled by 71,..., 7m, and at most one 
edge between each pair of vertices. The product of indicator 
functions is over edges G G. It constrains the set of 

polymers 71, ...,7a/ to intersect according to the structure of 
G. In this expansion one sums over an infinite number of 
terms so it is important to address the question of convergence. 
A criterion which ensures the convergence of the expansion 



uniformly in system size n (and thus ensures convergence in 
the infinite size limit) is 

+00 

To illustrate the use of the polymer expansion in a simple 
case, consider a vertex model at high temperature defined by 

faiaaa) = i(l + tanhJ, J| aab)ei'''^"'-' 

beda 

where Ja and hab are e M with sup^^^y = J << 1 and 
sup^jjg^; hab < h < +00. For J is small enough the BP (|5]l 
equations have a unique fixed point solution ||6l. We call ff^ 
this fixed point. The subscript n indicates (with some abuse of 
notation) that this fixed point depends on the finite instance, 
that is, the graph F, and Ja, hab- For the activities of the 
polymers, computed at the fixed point, we have the bounds 
\K{'fi)\ < (2J)I''''I. Moreover the number of polymers 79a 
is (for each a) at most e'^'*''''! with > a numerical constant 
depending only on d. Using also that the smallest polymer 
must have I7I > 3, it is then easily shown that the left hand 
side of (O is 0{J^) << 1. By standard methods m one can 
then estimate the sum over M in O term by term, which 
yields 

l/corr m I < (1 + 0( J^))^ E E(2'^)'''^'''' (9) 
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Proposition 3.1: For J < Jo{h) small enough, we have 

lim„^ + oo E[/corr(^i)] = 0. 

Proof idea: From ® E[|/eorr (O I] < (1 + 
0(J^))EE^3o(2J)lTlelTl]. Here a is any specified node in 
the graph. In order to conclude it suffices to use the fact that on 
a random c?-regular graph, with probability 1— o„(l), polymers 
have a size I7I > a^lnn (a^ > a positive numerical 
constant). ■ 

IV. Cycle LDPC codes over the BSC 

Random d-regular graphs are equivalent to the LDPC(2, d) 
ensemble of cycle codes. Code bits Xab = 0, 1 are attached 
to the edges ab £ E. In the spin language bits are aab = ±1 
and the parity check constraints are HbeSa '^"■b ^ ^- ^'-'^ 
niteness we assume transmission over the BSC(p), p G [0, 5]. 
Without loss of generality one can assume that the transmitted 
word is ( !,...,!) so that MAP decoding is based on the 
posterior distribution ([1]) with 

faiaoa) = ^(1 + n ^-fc) n e^'"""""- (10) 

b^da b^da 

Here hab is the half-log-likelihood for the bit (Tab — ±1, based 
on the channel output. The Shannon conditional input-output 
entropy and free energy are essentially equivalent, and related 
by the simple formula, 

ii7(X|y) = E^[/„(M]-i^lni^ (11) 



where E^- is the average over channel outputs (or the log- 
UkeUhood vector). 

We interested in the high noise regime where p is close to 
1/2. Therefore we seek solutions of the BP equations such 
that supaftg^; I /lab I < h where ft, > is a fixed small number 
We assume that for h small enough there exists a fixed point 
of the BP equations for each finite instanc^E We denote it ff^ 
as before. Note that assuming its unicity is not needed. 

Proposition 4.1: Assuming the existence of a fixed point 
fj^ of the BP equations for h small enough, we have 

lim„^+ooE[ilnZeorr(C)] =0. 

In view of (fTTI) the proposition implies that the average 
conditional entropy is given by the average of the Bethe 
expression computed at the fixed point (for |p — i| << 1). 

In order to prove the proposition we will use the identity 



Now note that for polymers 6(7) < X]f=2^('^ ^ *)"i(7) ^ 
d^'^~2 ni{j). Therefore we deduce thanks to ( fTSl l that 
with high probability X)i'=2^ "-^(t) — O.I8I7I and K{'-f) < 
(2/i)0.i8|7l for <- I This is sufficient to control the 
convergence criterion, and achieve the proof of this lemma 
by methods similarly to the high temperature case. ■ 
Lemma 4.3: Fix e > 0. Then 

P[V7 C r : < < e2-] > 1 - io„(l). 

The proof uses rather trivial bounds on the partition functions. 
We omit the details. 

Lemma 4.4: Fix 5 > 0. There exists a numerical constant 
C > and such that for h small enough 
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where 
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[]l(7,n7, =0) (13) 



and Zp{ff) = Zp{ff\^). This identity is derived by splitting the 
sum over 71, 7m in ©, into a sum where all polymers are 
small (Vi, |7i| < n/2), and a sum where there exists at least 
one large polymer (3i, > ri/2); and by noting that when 
there exists a large polymer it has to be unique. 
We will need three lemmas. 

Lemma 4.2: For h small enough we have 
lim„^+ooE[ilnZp(^)] =0. 

Sketch of Proof : It is possible to estimate the activities 
computed at the fixed point. 



\K{j)\<{l~a,-h'r(''-)l[{a,h''-^) 



(14) 
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Here < < 1, and ai > 1, i = 2, ...^d — 1 are fixed 
numerical constants (that we can take close to 1). The 71^(7) 
denotes the number of nodes of degree i in the polymer 7. 
Estimate (fT4l i is essentially optimal for h small, as can be 
checked by Taylor expanding -^'(7) in powers of hab- Hard 
constraints manifest themselves in the factor (1 — a^f ft^)"''*-^^ 
which is not small enough to compensate the entropic term 
gCdbl in the convergence criterion. However for polymers of 
size I7I < 2: vve can use expander arguments to circumvent 
this problem. Let e{g) the set of edges in E connecting g to 
y \ g. We say that T is an expander if for all 17 C F with 
I5I < f we have |e(.g)| > K\g\. For all d > 3 H 

P[r is an expander with n = 0.18 d] = 1 - On(l) (15) 

'This assumption can be relaxed by softening the hard constraint in jlOt 
and using existence results (7). Indeed all our estimates are uniform in the 
softening parameter We omit this discussion here due to lack of space. 
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This inequality is a fortiori valid for g's replaced by 7's in the 
sum. 

Sketch of Proof: We denote by /C„ the complete graph 
with n vertices. By Markov's inequality. 
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gCK,^ ^ i=2 

(17) 

Consider graphs g with ni{g), i = 2, ...,d fixed. Mackay ||9l 
provides a bound for the probabilit}0 ¥[g C F] of finding 
a particular subgraph into a regular graph F. Namely for 

iEt2*"^(5)+2d2<f, 



CF] < 



nt2 



The number of subgraphs of /C„ with given ni{g) is 



(18) 



(19) 



Replacing (fTSl l in ( fTTl l. using ( fT9l ). setting Xi — and 
performing an asymptotic calculation for n large, we show 



(here x = {x2, ...,Xd) and A = {x\^ < J^i 
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(20) 
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-Here [m]i = m{m — l)...(m — i + 1). 



The large n behavior of the integral asymptotic is controlled 
by fn{x), and gn{x) gives sub-dominant contributions that do 
not concern us here. We have 

^ 1 ^ 

i=2 1=2 i=2 VJ 

(1 - X: -0 ln(l - E - - — ) - — ) 

i=2 i=2 

Iv-- 2r\ , ,r 1^. 2r\ 
+ (9-9^*^^-— )-M;j-:^E*^'- (21) 



1=2 i=2 

For /i small enough, in the domain A, the exponent in (l20l l is 




/(i2,i3)+.i2!n('") + i3'n(l-i''*)""--- 



Fig. 2. The exponent in )20t for d = 3, for /i small enough, is strictly 
negative in the domain A. Its maximum at X2 = 0, 2:3 = 1 is 0(/i^). 

Strictly negative and attains its maximum at the corner point 
X2 = ■ ■ ■ = Xd-1 = 0, a:;^ = 1. At this point it is equal to 
ln(l — Ud^h?) which allows to conclude ( fT6b . ■ 
We are now in a position to prove proposition 14.11 

Proof of proposition \4.1\ In view of ( fT2l l. we must show 
that for h small enough, 
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o„(l). (22) 



Call Ir the event 



l7l>«/2 „ . 

E mi)\^4^<c 
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where is a positive constant that will be adjusted later on. 
We spht the expectation in two terms A + B hy conditioning 
over Ii^ and its complement and estimate each contribution. 
For the first contribution, using | ln(l + a;)| < | ln(l — |a;|)| 
for \x\ < 1, A < i|ln(l - C)|P[/c] < ^|ln(l - 01- For 
the second contribution we have to estimate P [/,?]. The events 

{V7 C r : <^< e^-n and {E[Tr"^' \Kh)\ < 

6} imply Ige^"^- Therefore /|g2nt implies the union of the 
complementary events, so that applying the union bound 
together with lemmas 14.31 and 14.41 

P[/L-=] < ye-""4'^^ + io„(l). 



Now suppose for a moment that there exist a positive constant 
independent of n such that 
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Then B < C"P[/^]. Setting C 
imply 
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Je^"*^, the above arguments 
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adjgh (lemmas 



B — o„(l), which proves 



We are free to choose 6 = e 
|43]I13] hold) and this choice A 

It remains to justify ( |23] |. From the convergence of the 
polymer expansion we deduce that ^ In Zpifj^^) is bounded 
uniformly in n. From ([1]), (fTOl l we easily show that -InZr < 
In 2 + ^h. In the high noise regime the BP messages are 
bounded so that from (O we deduce that i In ZBcthc(f?^) is 
bounded by a constant independent of n. Finally the triangle 
inequality implies that i| lnZr-lnZBothc(»/^)-ln^p(?/J^)| is 
bounded uniformly in n. This is precisely the statement (|23]) . 



V. Conclusion 

The approach is quite general and can hopefully be general- 
ized to standard irregular LDPC codes with bounded degrees 
and binary-input memoryless output-symmetric channels with 
bounded log-likelihood variables. This will be the subject of 
future work. 
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